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2.0  EXECUTIVE  SUMMARY 

The  group  addressed  the  issue  of  information  and  data  source  discovery  in  a  real-time  NATO  or  coalition 
operational  environment.  The  group  analyzed  a  conceptual  approach  to  the  design  of  an  interactive 
visualization  system  to  be  used  for  data  resource  discovery.  Needs  and  requirements  are  discussed  through 
a  process  of  answering  (in  part)  six  key  questions  that  would  guide  system  design.  The  group  chose  a  use- 
case  for  illustration.  The  group  concludes  by  recommending  that  the  issue  of  data  resource  discovery, 
understanding,  and  availability  in  a  NATO  operation  be  further  studied. 


3.0  INTRODUCTION 

3.1  NATO  Operational  Problem  Statement 

In  a  multinational  coalition  operational  environment,  the  effectiveness  of  a  commander’s  decision-making 
can  be  impaired  during  critical  real-time  planning  activities  by  the  lack  of  knowledge  of  information 
resources.  High-tempo  decision  makers  require  an  awareness  and  understanding  of  the  existence, 
availability,  and  quality  of  time-dependent  information  resources.  These  resources  can  include  national, 
coalition,  theater,  organic,  or  global  sensors,  systems  and  platforms,  or  other  data  sources  (eg.  open  press, 
media,  www,  . . .  etc.). 

From  the  Operational  Commander  (User)  - 

“...NATO  nations  in  an  operational  context,  often  do  not  adequately  share  national 
information  assets  and  products  in  an  efficient  and  effective  manner...” 


Paper  presented  at  the  RTO  1ST  Workshop  on  " Massive  Military  Data  Fusion  and  Visualisation:  Users  Talk 
with  Developers",  held  in  Halden,  Norway,  10-13  September  2002,  and  published  in  RTO-MP-105. 
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3.2  Question  Addressed  by  IST-036/RWS-005  Syndicate  2 

How  might  a  visualization  system  assist  in  the  understanding  of  the  existence,  availability,  and  quality  of 
information  resources  in  a  distributed  NATO  or  coalition  operational  environment? 

3.3  Applicability  to  Counter-Terrorism  Application 

Counter-terrorism  activities  often  involve  multi-national  or  coalition  efforts.  Additionally,  non-military 
resources  such  as  local  law  enforcement  and  emergency  response  personnel  may  be  utilized,  each  having 
its  own  as  well  as  shared  information  and  data  resources.  Access  and  understanding  of  these  resources  can 
play  a  vital  role  in  a  commander’s  operational  effectives  and  situational  awareness. 


4.0  PROBLEM  ANALYSIS 

In  order  to  conduct  an  analysis  of  this  problem,  the  group  chose  to  follow  the  VisTG1  model  of 
visualization  system  design.  This  model  was  chosen  in  order  to  use  the  posed  questions  as  guidance  in  the 
analysis  process.  The  group  further  developed  a  brief  straw-man  and  operational  scenario,  in  order  to 
illustrate  the  key  features  and  issues  to  be  addressed. 

4.1  Example  Use  Case 

To  assist  the  analysis  and  understanding  of  this  problem,  the  group  explored  the  potential  situation  of  a 
downed  pilot  during  a  tactical  mission  tasking.  The  tactical  operational  commander  needs  to  obtain  the 
necessary  information  in  order  to  plan  and  execute  an  extraction  operation.  Following  is  a  possible  chain- 
of-events  as  they  would  occur  on  a  commander’s  data. 

1)  Pilot  goes  down. 

2)  Transponder  on  aircraft  is  activated. 

3)  Table  console  (TC)  in  command  and  control  (C&C)  flashes  red  dot. 

4)  Commander  zooms  TC  into  pilot  location  to  view  terrain/image  model  at  fine  resolution. 

5)  J2  and  J3  initiate  a  data  availability  request  for  area  of  interest  (AOI)  around  the  pilot. 

6)  Data  list  is  retrieved  from  metadatabase  that  is  continuously  updated  by  Allies.  Such  a  list  might 
include: 


Source  Platform 

Available  Data 

ETA 

NIC 

UAV 

High  Resolution  IR 

5  min 

US-12 

Platoon 

Visual 

30  min 

NOR-2 

Satellite 

1  m  Multispectral 

3  hours 

CAN-3 

Special  Forces 

Medical  Condition 

1  hour 

UK-1 

F-16 

Sighting 

- 1 0  min 

FRA-4 

7)  J2  and  J3  chose  from  the  above-noted  list  then  initiate  AOI  data  requests  to  all  relevant  National 
Intelligence  Centers  (NIC’s)  including  commander’s  own  NIC. 


1  The  VisTG  Model  identifies  a  hierarchical,  nested  processing  loop  for  visualization.  The  model  poses  six  key  questions  to 
assist  in  the  design  of  a  visualization  system. 
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8)  Metadatabase  indicates  probable  time  of  data  retrieval.  In  addition,  answers  are  received  from 
NIC’s  for  non-standard  data  requests. 

9)  As  data  is  received  or  retrieved,  update  the  TC  and  provide  personal  reports  for  J2  and  J3  to  the 
Commander.  Objects  of  interest  would  be  displayed  as  icons  that  are  easily  identified  at  a  glance. 

10)  Have  TC  report  the  percentage  of  data  downloaded  from  each  NIC  and  expected  time  of  receipt 
of  remaining  data.  These  might  be  displayed  on  the  TC  as  NIC-specific  information  bars. 

1 1)  Have  preset  datasets  (groups  of  data  types)  selectively  available  at  the  touch  of  a  button.  These 
could  be  pushed  by  the  commander  or  requested  from  the  J2  or  J3.  The  dataset  would  also  be 
editable  to  include  more  or  less  sets. 

12)  Display  the  “probability  of  detection”  or  “probability  missing  data”  for  any  variable  the 
commander  specifies. 

13)  Predict  and  display  the  enemy  forces’  situational  awareness  of  AOI. 

14)  Enable  ability  to  gain  control  of,  and  task,  mobile  sensors. 

4.2  Using  the  VisTG  Reference  Model  for  Design 

Several  times  during  the  workshop  it  was  mentioned  that  one  of  the  problems  with  coalition  operations  is 
that  nations  do  not  necessarily  make  pass  to  the  coalition  command  all  the  information  they  have.  Some  of 
this  information  might  be  made  available  on  request,  some  might  be  made  available  after  being  sanitized, 
and  some  might  not  be  made  available  outside  the  national  command.  Furthermore,  the  command  staff 
may  find  for  particular  situations  that  unconventional  information  sources  are  both  available  and  useful. 

Accordingly,  an  appropriate  task  for  the  Syndicate  seemed  to  be  to  suggest  a  visuaalisation  system  that 
would  assist  a  commander  and  the  command  staff  to  determine  what  sources  might  be  able  to  provide 
what  kinds  of  information  relevant  to  the  situation  of  immediate  interest.  It  was  assumed  that  very  many 
different  kinds  of  data  source  might  be  available,  both  official  and  unofficial,  and  that  the  nature  of  all  of 
them  would  be  contained  in  some  kind  of  a  dataspace  accessible  to  the  user  (the  command  staff).  There 
would  be  too  many  possible  data  sources  for  any  staff  officer  to  be  able  to  remember  all  of  them  reliably. 

The  method  chosen  for  the  development  was  the  VisTG  Reference  Model  illustrated  in  Figure  1.  To  make 
the  example  concrete,  a  use-case  was  chosen,  in  which  a  plan  was  to  be  developed  for  recovering  a 
downed  pilot. 
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Figure  1 


The  VisTG  Reference  model  treats  visualisation  as  one  of  two  routes  to  understanding,  the  other  being 
analysis.  The  user  wants  to  understand  and  to  act  upon  some  dataspace  that  contains  information  about  the 
world  of  interest.  Visualisation  may  be  considered  similar  to  intuition,  which  both  supports  and  is 
supported  by  analysis. 

The  Reference  Model  does  not  explicitly  address  the  analytic  process,  which  has  different  presentation 
requirements  than  does  visualisation.  In  particular,  analysis  ordinarily  deals  in  individuated  entities  and 
their  relationships,  whereas  visualisation  is  more  concerned  with  patterns  in  an  extended  context. 

Analysis  is  impeded  by  clutter  in  the  presentation,  whereas  visualisation  may  be  aided  by  the  same  clutter, 
clutter  which  is  similar  to  that  with  which  every  person  is  confronted  every  moment  of  the  day.  To  put  it 
crudely,  visualisation  depends  heavily  on  context,  whereas  analysis  depends  on  individuation  and  is 
normally  context-free. 


4.3  Design  of  a  Data-Source  Discovery  System 

The  design  process  using  the  VisTG  Reference  Model  proceeds  in  stages.  The  model  assumes  a  series  of 
nested  loops,  each  based  around  the  achievement  of  some  purpose,  or  goal,  and  the  perceptions  needed  by 
the  user  if  progress  toward  achieving  that  goal  is  to  be  assessed.  At  any  one  level,  there  may  be  several 
parallel  loops,  and  any  one  of  these  may  simultaneously  serve  more  than  one  purpose  at  a  higher  (outer) 
loop  level.  For  each  of  these  many  loops,  the  method  identifies  six  basic  questions,  which  can  be 
summarized: 

1)  What  is  the  purpose  of  the  loop? 

2)  What  does  the  user  need  to  perceive  if  the  purpose  is  to  be  achieved? 

3)  What  does  the  user  need  to  be  able  to  do  to  achieve  the  desired  perception? 

4)  What  impediments  might  detract  from  the  user’s  ability  to  perceive  (including  lack  of  user 
training)? 


S2  -4 


RTO-MP-105 


Data  Source  Discovery  in  Coalition  Operations 


5)  What  impediments  might  reduce  the  user’s  ability  to  act  to  affect  the  necessary  perception 
(including  lack  of  user  training)? 

6)  What  provisions  might  be  available  for  alerting  the  user  to  portions  of  the  dataspace  potentially 
relevant  to  the  purpose  of  the  loop? 

Outer  (Level  1)  Loop 

The  first  stage  of  the  design  process,  then,  must  be  to  assert  the  purpose  that  a  given  loop  is  to  serve.  In  the 
problem  at  hand,  the  system  to  be  designed  is  supposed  to  help  a  command  staff  (including  the 
commander)  to  be  able  to  understand  those  data  sources  that  might  reasonably  be  expected  to  have,  and  to 
be  willing  to  provide,  data  relevant  to  the  situation  of  immediate  interest,  including  understanding  of  their 
probable  reliability,  relevance,  and  latency  (time  to  deliver  the  information). 

Translating  these  requirements  to  the  use-case,  relevant  data  sources  include  those  that  could  potentially 
supply  information  about: 

•  The  location  of  the  downed  aircraft  and  its  pilot 

•  The  pilot’s  state  of  health 

•  The  character  and  socio-political  environment  of  the  people  in  the  region 

•  The  location,  numbers,  identity,  and  movement  of  nearby  blue,  red  and  orange  forces 

•  The  local  physical  environment,  such  as  terrain,  weather,  trafficability,  visibility,  and  so  forth. 

The  answer  to  Question  1  for  the  outer  loop  (i.e.,  Q  1.1),  What  is  the  purpose  of  the  system,  then,  is  that 
the  system  is  to  help  the  command  staff  to  distinguish  from  a  potentially  very  large  number  of  official  and 
unofficial  data  sources  those  that  could  potentially  supply  any  of  the  desired  kinds  of  information,  and  to 
prioritize  among  the  potentially  relevant  ones  those  that  would  be  most  likely  to  be  useful  within  the  time 
frame  available  for  the  operation. 

Question  1 .2,  What  does  the  user  need  to  perceive,  is  about  the  relationship  of  the  sources  to  the  necessary 
data.  The  data  requirement  information  is  paramount,  the  source  is  a  means  to  acquire  it.  This  means  that 
the  user  does  NOT  need  to  perceive  available  data  as  an  attribute  of  the  set  of  potential  data  sources. 
Rather,  sources  that  could  provide  data  such  as  location,  politico-social  climate,  or  local  weather,  should 
be  presented  as  attributes  of  the  data  type,  however  that  presentation  is  instantiated.  The  user  knows  the 
required  data,  and  needs  to  be  able  to  use  that  knowledge  as  an  “index”  into  the  potentially  large  set  of 
possible  data  sources.  The  system  must  be  able  to  present  the  data  so  that  the  user  can  see  not  only  which 
sources  can  provide  which  specific  kinds  of  data,  but  also  the  likelihood  of  getting  the  information,  its 
probable  reliability,  and  the  probable  delay  (latency)  before  the  information  is  made  available. 

Question  1.3,  What  does  the  user  need  in  order  to  be  able  to  act  to  achieve  the  desired  perception,  is  about 
the  user’s  ability  to  let  the  system  know  what  kinds  of  data  are  desired,  so  that  the  system  can  search  its 
dataspace  to  determine  what  sources  might  be  able  to  provide  them.  For  example,  if  the  user  wants  to 
contact  a  known  friendly  person  in  a  town  near  the  downed  pilot,  a  local  telephone  directory  would  be  a 
useful  data  source.  A  taskable  UAV  might  be  available  to  provide  detailed  terrain  information,  and  one 
might  actually  be  near  the  place  in  question,  though  tasked  for  some  other  purpose.  A  satellite  image  or  a 
meteorological  officer  might  be  useful  data  sources  if  the  required  information  is  the  local  weather  in  the 
next  few  hours.  The  same  satellite  image  might  be  a  useful  data  source  for  other  aspects  of  the  physical 
environment.  SIGINT  and  HUMINT  sources  collated  by  coalition  and  national  intelligence  cells  might 
provide  information  about  force  movements,  but  some  national  cells  might  divulge  only  sanitized 
versions,  and  then  only  after  appreciable  delay. 
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The  user’s  actions  need  to  be  able  to  give  the  system  enough  information  that  the  system  is  able  to  present 
to  the  user  what  is  necessary  to  perceive.  The  user  must  be  able  to  specify,  for  example,  limits  on  latency. 
Under  some  circumstances,  retrieving  the  downed  pilot  might  require  no  more  than  asking  the  local  mayor 
to  put  the  pilot  on  a  bus,  with  a  promise  to  refund  the  ticket  price.  Under  others,  a  delay  of  30  minutes 
might  make  the  difference  between  success  and  failure  of  the  retrieval  mission  in  hostile  territory.  The  use 
has  to  be  able  to  let  the  system  know  the  degree  to  which  latency  matters,  just  as  much  as  to  let  the  system 
know  what  kinds  of  data  are  required. 

Question  1.4  What  impediments  might  detract  from  the  user’s  ability  to  perceive  what  is  necessary  is 
about  both  external  impediments,  such  as  the  uncertain  willingness  of  National  Intelligence  Cells  to 
release  information  they  may  be  known  to  have,  and  internal  impediments  such  as  the  user’s  lack  of 
knowledge  that  certain  kinds  of  data  or  sources  of  data  might  be  accessible  or  useful.  The  latter  component 
may  be  characterized  as  possible  lack  of  training  or  experience.  The  former,  however,  can  be  treated 
directly,  and  answered,  at  least  in  part,  by  saying  that  some  sources  may  not  identify  the  kinds  of  data  they 
are  willing  to  provide  on  request.  Consequently,  the  appropriate  linkages  would  not  occur  in  the  database 
and  could  not  be  presented  by  the  system. 

Question  1.5  What  impediments  might  detract  from  the  user’s  ability  to  act  to  generate  the  desired 
perception  is  about  the  user’s  ability  to  get  out  of  the  database  the  displays  that  would  show  the 
immediately  useful  sources,  the  usefulness  of  the  data  they  could  provide,  and  the  means  to  acquire  the 
information  from  the  useful  sources.  It  is  a  question  about  user  control  of  the  content  of  the  presentation 
displays  (sometimes  erroneously  called  the  “visualisations”).  An  example  of  a  potential  impediment  might 
be  an  inability  to  specify  the  relevance  of  a  sufficient  range  of  data  attributes,  either  because  of  restrictions 
in  available  control  menus  or  data  glossaries,  or  because  of  inadequate  user  training  in  the  use  of  a 
complex  interface.  In  the  use-case,  for  example,  the  system  might  make  it  difficult  for  the  command  staff 
to  specify  that  they  needed  to  ascertain  how  long  it  would  take  to  acquire  imagery  from  a  UAV  that  might 
already  be  in  the  vicinity  or  that  might  have  to  be  tasked  to  take  off  on  a  specific  mission  related  to  the 
retrieval  of  tha  downed  pilot.  This  infonnation  might  affect  the  relevance  level  of  other  possible  sources  of 
similar  data. 

Question  1.6  What  provision  is  there  for  alerting  the  user  to  potentially  useful  regions  of  the  dataspace 
concerns  autonomous  actions  performed  by  the  system  under  general  rather  than  specific  control  of  the 
user.  The  user  (or  system  designer)  may  set  up  criteria  for  determining  what  patterns  in  the  dataspace 
warrant  being  labelled  “potentially  useful”,  but  the  system  does  the  labelling  independently  of  immediate 
user  control.  Alerting  occurs  when  the  system  affects  the  presentation  so  as  to  draw  the  user’s  attention, 
however  momentarily,  to  the  “potentially  useful”  aspect  of  the  dataspace.  Typically,  the  labelling  will  be 
affected  by  prebuilt  scenarios.  For  the  use-case  of  a  downed  pilot,  the  system  might  be  set  up  to  highlight 
data  sources  related  to  the  attributes  mentioned  above:  location,  terrain,  weather,  pilot’s  health,  local 
socio-political  environment,  force  dispositions,  etc.  For  other  scenarios,  other  constellations  of  data 
sources  might  be  highlighted. 

Engine  (Level  2)  Loops 

The  questions  discussed  above  are  all  cast  in  terms  of  the  purpose  of  the  system  as  a  whole,  and  as  a 
group,  they  specify  the  requirements  and  illustrate  some  possible  pitfalls  to  be  avoided  in  the  design. 
The  design  process  builds  on  the  answers  to  the  Level  1  questions,  using  them  as  partial  specifications  for 
questions  that  define  Level  2  loops. 

Generically,  any  visualisation  system  has  four  kinds  of  engine  loops  at  level  2: 

•  Navigation  Engines,  which  allow  the  user  to  move  around  the  dataspace, 

•  Data  Selection  Engines,  which  allow  the  user  to  choose  subsets  of  the  data  for  manipulation, 
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•  Algorithm  Selection  Engines,  which  allow  the  user  to  determine  how  the  selected  data  subsets  are 
to  be  manipulated,  and 

•  Algorithm  Execution  Engines,  which  do  the  actual  manipulation  of  the  data,  including  the 
preparation  of  presentations  such  as  2-D  and  3-D  displays,  textual  and  tabular  representations, 
or  auditory  or  haptic  displays. 

In  a  detailed  design,  each  of  the  six  questions  should  be  addressed  for  each  engine  in  each  of  the  four 
classes  of  engines.  For  the  present  purposes,  it  may  suffice  to  illustrate  some  examples  and  suggest  the 
implications  of  the  answers  for  the  presentation-level  systems  (e.g.  screen,  keyboards,  etc.). 

Navigation  Engines  allow  the  User  to  examine  different  parts  of  the  dataspace.  They  tend  to  work  closely 
with  Data  Selection  Engines.  Indeed,  data  selection  can  be  an  aspect  of  navigation,  in  that  it  can  refine 
whole  sections  of  the  dataspace,  in  effect  reducing  the  volume  in  which  navigation  might  occur.  The  user 
may  navigate  (alter  the  viewable  part  of  the  dataspace)  and  then  select  within  the  displayed  parts  of  the 
dataspace,  or  may  first  select  using  definable  attributes,  and  then  navigate  within  the  selected  subset  of  the 
data.  In  the  end,  however,  the  result  is  a  data  subset  on  which  selected  algorithms  may  operate,  both  to 
change  the  content  of  the  dataspace  and  to  prepare  for  the  actual  user  presentation  of  the  computed  results. 
The  six  wuestions  can  be  applied  to  each  engine  within  each  class.  Here,  for  simplicity,  the  system  will  be 
treated  as  if  there  were  only  one  engine  in  each  class. 

Navigation  engine(s): 

Q2.1.nav  (Purpose)  The  user  needs  to  be  able  to  navigate  so  that  relevant  data  sources  are  viewable. 

Q2.2nav  (Perceptual  requirement)  The  user  needs  to  be  able  to  see  the  relevance  of  data  sources 
brought  into  view,  to  be  able  to  see  where  to  navigate  so  as  to  view  other  data  sources,  and  to  be  able 
to  see  the  functions  of  any  control  mechanisms  provided  to  change  the  view. 

Q2.3nav  (Ability  to  act)  Controls  must  be  available  to  allow  the  user  to  affect  the  data  sources 
displayed,  in  such  a  way  that  the  various  attributes  of  the  data  sources  may  be  used  to  affect  which 
ones  are  actually  displayed. 

Q2.4nav  (Possible  impediments  to  perception)  Inability  to  see  the  functions  of  available  controls 
(either  because  of  presentation  design  or  because  of  lack  of  training);  inability  to  see  where  in  the 
dataspace  might  be  a  useful  region  to  examine;  inability  to  see  the  relevance  of  potential  data  sources 
(because  of  presentation  design  flaws  or  lack  of  training). 

Q2.5nav  (Possible  impediments  to  action)  Failure  of  design  to  provide  appropriate  navigation 
controls.  Complexity  of  control  mechanisms  requiring  use  of  controls  not  simultaneously  accessible. 

Q2.6nav  (Potential  Alerting  systems)  Agents  might  be  developed  to  indicate  useful  data  sources  for  a 
variety  of  standard  scenarios,  and  thereby  to  alert  the  user  to  possibilities  that  might  not  be 
immediately  obvious. 

Data  Selection  Engine(s): 

Q2.1DS  (Purpose)  To  allow  the  user  to  select  those  data  sources  most  likely  to  be  immediately  useful 
(and  in  a  broader  appreciation  of  the  system  in  use,  thereby  to  allow  the  user  to  begin  the  process  of 
acquiring  the  data  from  the  selected  sources). 

Q2.2DS  (Perceptual  Requirement)  To  allow  the  user  to  perceive  which  sources  are  currently  selected, 
and  to  perceive  how  to  alter  the  selected  subset  by  addition  or  removal  (i.e.  to  perceive  the  means  of 
controlling  the  selection). 

Q2.3DS  (Ability  to  Act)  Means  must  be  provided  for  the  user  to  add  to  or  exclude  from  the  selection 
by  data  attribute,  by  source  identity  or  attribute  (e.g.  nationality,  latency,  or  reliability),  by  location 
within  the  display,  or  by  other  suitable  attribute. 
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Q2.4DS  (Possible  impediments  to  necessary  perceptions)  Inability  to  distinguish  selected  from  non- 
selected  data  sources  (e.g.  colour  blindness);  Inability  to  perceive  data  attributes  relevant  to  selection; 
Failure  in  the  design  to  make  apparent  the  functions  of  selection  controls. 

Q2.5DS  (Possible  impediments  to  action)  Failure  to  provide  means  for  selection  by  potentially  useful 
source  or  data  attributes.  Lack  of  training  in  the  use  of  selection  mechanisms  actually  provided. 

Q2.6DS  (Alerting)  Ambiguities  of  selection  could  be  represented  and  opportunity  for  disambiguation 
provided. 

There  is  no  need  here  to  analyze  the  possible  loops  for  Algorithm  selection  and  execution  loops.  Among 
the  algorithms  might  be  those  that  enable  the  user  actually  to  begin  the  process  of  acquiring  the  data  from 
the  source,  such  as  by  initiating  the  tasking  of  a  UAV,  by  looking  up  a  phone  book  for  the  locality  of  the 
downed  pilot,  by  generating  a  formal  request  for  data  from  a  national  NIC,  and  so  forth.  All  of  these 
represent  operations  on  the  data,  and  have  implications  for  the  actual  presentations  generated  by  the 
system.  However,  the  question  at  issue  is  the  discovery  of  the  set  of  most  useful  data  sources  for  the  user’s 
immediate  problem,  which,  in  the  use-case,  is  to  generate  a  plan  for  the  recovery  of  the  downed  pilot. 
The  actual  generation  of  the  plan  is  outside  the  scope  of  the  present  discussion. 

The  answers  to  the  Engine-level  questions  have  implications  for  the  physical  design  of  the  presentations 
and  the  control  input  devices.  For  example,  the  answers  to  Q2.2nav  and  Q2.2DS  suggest  that  at  least  two 
kinds  of  display  would  be  useful  in  the  use-case  example,  one  map-based,  to  allow  theuser  to  limit  the 
range  of  data  sources  to  those  potentially  able  to  provide  information  relevant  to  the  area  of  interest,  and 
the  other  somewhat  VITA-like  in  which  data  types  and  sources  are  cross-linked  so  that  sources  with  much 
coordinated  relevant  data  could  be  given  selection  priority  (see  Annex). 

Likewise,  the  answers  to  the  Q2.3  questions  indicate  the  need  for  clear  and  clearly  labelled  controls  for 
exploding  aggregated  data  sources,  and  for  quasi-spatial  navigation  within  a  displayed  2-D  or  3-D 
presentation  space  (especially  with  map  or  VITA-like  displays).  For  data  selection,  controls  based  on  area 
or  volume  selection  within  the  presentation  space,  or  on  linguistically  labelled  data  or  source  attributes 
marking  selection  boundaries  numerically  or  using  sliders,  would  seem  to  be  necessary. 

The  detailed  design  of  the  presentations  forms  the  third  loop  level,  and  as  with  the  other  two  levels,  the 
purposes  of  the  presentations  are  determined  by  the  answers  to  the  questions  at  the  level  immediately 
above.  Some  of  the  more  obvious  implications  are  indicated  above,  but  for  a  full  design,  each  of  the  level 
2  answers  needs  to  be  seen  as  a  generator  of  purpose  for  one  or  more  elements  of  the  presentation  displays 
and  of  the  control  input  mechanisms.  Once  those  purposes  have  been  defined,  the  actual  presentations  and 
the  means  whereby  the  user  can  navigate  within  them  and  move  among  them  can  be  specified  using  the 
same  pattern  of  five  questions  (the  sixth  being  Q  1,  about  Purpose,  which  has  already  been  answered). 

At  each  level,  when  there  are  parallel  loops,  the  designer  must  ask  about  the  possibilities  for  interference 
among  them.  If  the  user  is  paying  attention  to  one,  does  that  detract  from  the  immediate  usefulness  of 
another?  Does  the  provision  of  multiple  perceptual  possibilities  create  confusion  or  context?  Are  the 
required  actions  mutually  incompatible?  Is  the  question  of  the  moment  one  of  precision  or  of  pattern? 
All  such  questions  must  be  resolved,  whether  by  design  or  by  default,  when  a  system  is  finally  built. 


Annex:  The  VITA  Presentation 

The  VITA  engine  suite  (Jacobson,  McIntyre  and  Romet,  Workshop  ppresentation  Session  6) 
was  developed  in  the  context  of  discovering  patterns  of  concepts  within  Web  pages,  but  is  not  limited  to 
that  area  of  application.  It  contains  both  Data  Selection  and  Algorithm  Execution  engines,  and  Algorithm 
Selection  is  to  some  extent  also  incoiporated  in  the  form  of  prior  user  control. 
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In  the  VITA  original  design,  one  or  more  “Query”  instances  are  submitted  to  search  engines.  Each  Query 
contains  a  set  of  concepts  (keywords,  in  the  initial  implementation),  and  the  search  engines  return  for 
display  Web  pages  that  contain  at  least  a  threshold  number  of  concepts.  In  the  3-D  VITA  display, 
the  Queries  are  displayed  as  nodes  in  one  plane,  the  concepts  as  nodes  in  a  second  plane,  and  the  Web 
pages  that  pass  the  threshold  test  as  nodes  in  a  third  plane.  Concept  nodes  are  shown  as  linked  to  Query 
nodes  on  the  one  side,  and  to  Web  page  nodes  on  the  other.  The  placement  of  concept  and  Web  page 
nodes  within  their  respective  planes  is  controlled  by  a  self-organizing  process  that  locates  similar  entities 
near  each  other. 

In  the  context  of  a  data  source  discovery  process,  a  similar  display  might  substitute  Standard  Scenarios  for 
the  Queries,  Data  Requirements  for  the  Concepts,  and  Data  Sources  for  the  Web  pages.  The  user’s  eye 
would  tend  to  be  drawn  to  those  data  sources  most  relevant  for  the  scenario  at  hand.  If  more  than  one 
scenario  applied,  the  display  would  be  essentially  the  same,  except  that  the  “Query”  plane  would  be  more 
heavily  populated,  as  is  the  case  with  “standard”  VITA  when  displaying  the  combined  results  of  several 
independent  Queries. 

The  standard  VITA  display  allows  for  variable  representation  of  concept  and  page  nodes  according  to  their 
attributes,  such  as  whether  the  page  has  been  examined.  Using  a  VITA-like  display  for  Data  Source 
discovery,  valuable  attributes  might  affect  either  the  placement  or  the  representation  of  the  node,  or  both. 
For  example,  probable  latency  might  affect  the  transparency  or  some  shape-related  iconic  value  of  the  data 
source  representation,  or  might  affect  its  location  above  or  below  its  “natural”  representation  plane.  Or,  if 
the  user  were  given  the  ability  to  indicate  to  the  system  the  relative  importance  of  attributes  such  as 
latency  and  reliability,  the  data  sources  might  be  represented  in  colour  variations  that  represent  the 
important  attributes.  There  are  many  possibilities  for  enhancing  the  standard  VITA  presentation  to  take 
advantage  of  the  requirements  brought  out  by  the  “six-question”  analyses  generated  by  using  the  VisTG 
Reference  Model. 


5.0  RESEARCH  AND  DEVELOPMENT  ISSUES 

The  following  are  some  research  issues  that  should  be  addressed  to  achieve  the  desired  capability. 

1)  Information  translation/transformation  -  how  can  general  information  requests/queries  be 
translated  into  specific  requests  for  data?  For  example  if  a  user  wants  to  locate,  characterize  and 
identify  a  specific  target,  what  is  the  relationship  between  the  target  type  and  observable 
quantities?  Need  to  link: 

target  information  ->  observable  quantities  ->  sensors/sources 

Note  the  description  of  the  ontology-driven  sensor  independencies  (by  E.  Jungert)  is  pertinent 
here.  Special  challenges  here  include: 

•  situation  in  which  multiple  sources/sensors  might  work  in  concert  to  achieve  requisite 
information  (e.g.  observing  different  attributes  may  lead  to  identity  knowledge) 

•  understanding  how  to  relate  accuracy  requirements  to  sensor/source  performance. 

2)  Representation/Propagation  of  Uncertainty  -  How  can  we  link  information  requirements 
(confidence/uncertainty)  to  requirements  for  the  information  sources.  For  Example,  if  we  need  to 
know  the  location  of  a  target  within  one  meter,  what  does  this  imply  about  the  number  of 
measurements,  accuracy  of  angle  and  range  measurements,  etc.  A  more  difficult  issue  involves  the 
link  between  confidence  in  target  identity  and  observable  features.  How  can  we  show  what  is 
observable/knowable  and  what  cannot  be  observed  or  known. 
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3)  Visualization  of  source  availability/performance  -  How  can  we  readily  show  the  user  the 
availability  and  capability  of  sources.  It  is  one  thing,  for  example,  to  show  the  footprint  of  an 
airborne  based  image  sensor  (e.g.  to  show  what  the  sensor  could  “see”).  It  is  more  difficult  to 
understand  how  to  show  how  multiple  sensors  could  be  used  within  an  area  to  identify  a  target 
type  (Steinberg  &  Pack,  this  workshop). 

4)  Maintenance  of  a  Metadatabase  (data  about  data)  -  Data  about  what  sensor  data/information  is 
potentially  available  to  NATO  forces  needs  to  be  stored  in  a  common  format.  This  will  enable 
searches  and  queries  between  national  intelligence  centers.  This  database  will  be  maintained  and 
updated  by  all  involved  forces  as  new  data  is  made  available  or  existing  data  is  removed. 
The  database  can  include  reference  to  information  such  as  IR  images  owned  by  a  country,  the 
phone  number  to  a  mayor  in  a  nearby  city,  or  the  URL  of  a  site  containing  European  train  tables. 
Each  metadata  entry  would  include  at  least  the  following  information:  senor  type,  data  quality, 
area  covered,  time  of  acquisition,  time  to  become  available,  and  how  to  access  the  data. 


5)  How  to  discover  potential  data  sources  -  Incorporate  a  method  for  discovering  data  sources  that 
have  the  potential  of  delivering  relevant  data.  For  example,  the  fact  that  a  UAV  is  flying  within 
the  vicinity  of  the  AOI  needs  to  be  discovered  such  that  it  might  be  rerouted  to  fly  over  the  AOI. 

•  Nature  of  the  location  and  area  of  the  downed  aircraft.  What  are  the  physical  and 
environmental  factors? 

•  Potential  data  sources  -  organic  video,  satellite  imagery,  weather  maps 

•  Physiological  assessment  of  the  pilot.  State  of  vitals.  Mobility. 

•  Potential  data  sources  -  radio  comms,  cell  phone? 

•  Sociological  and  political  environment  in  the  area?  Status  of  civilian  crowds. 
Attitudes  towards  blue  forces.  State  of  local  police,  emergency  personnel  activities, 
etc. 

•  Potential  data  sources  -  Intelligence  packages,  video,  local  media,  on-site 
personnel  and  forces 

•  Status,  make-up,  threat  level,  and  locations  of  all  friendly,  hostile,  and  other  forces  in 
the  area. 

•  Potential  data  sources  -  organic  and/or  air/space  based  multi-national  sensors, 
Intelligence  packages 

•  Current  status  of  extraction  team  forces. 

•  Potential  data  sources  -  organic  command  and  control  system 

For  each  information  and  data  source,  it  is  also  required  to  know  the  timeliness  of  the  data,  its  reliability 
and  relevance  into  the  future  for  some  period  of  time.  Additionally,  what  is  the  time  required  to  get 
information  from  these  data  sources? 


6.0  CONCLUSIONS 

The  group  recommends  that  the  issue  of  the  knowledge  and  availability  of  all  relevant  and  potentially 
relevant  sources  of  data  and  information  by  NATO  commanders  in  a  multinational  or  coalition  operation 
be  further  studied  and  analyzed.  The  group  further  recommends  that  the  design  of  an  interactive 
visualization  system  that  would  assist  the  commander  in  information  and  data  resource  and  sensor 
discovery  be  explored. 
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Comment: 

In  Step  6  it  is  J2  and  J3  who  select  the  best  sensors. 

Question: 

Is  there  automation,  or  is  this  a  manual  process? 

Response: 

Automation  would  be  nice,  but  would  depend  on  the  sophistication  of  the  preset  databases.  The  idea  is  to 
get  the  requests  out  as  soon  as  possible. 
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•  The  problem  of  data  &  information  resource 
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•  Use  Case 

•  Recommendations 

•  Conclusion 


S2-2 


NATO  Operational  Problem 


•  ...In  a  multinational  coalition  operational 
environment,  the  effectiveness  of  a 
commander ’s  decision-making  can  be 
impaired  during  critical  real-time  planning 
activities  by  the  lack  of  knowledge  of 
information  resources . . . 
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Question  Addressed  by  IST- 
036/RWS-005  Syndicate  2 

How  might  a  visualization  system  assist  in 
the  understanding  of  the  existence, 
availability,  and  quality  of  information 
resources  in  a  distributed  NATO  or 
coalition  operational  environment? 


Problem  Analysis 


•  VisTG  Model  Analysis  -  Key  Questions 

What  is  the  user  trying  to  achieve  at  this  point  in  time? 

What  does  the  user  need  to  perceive? 

What  does  the  user  do  influence  the  goal? 

What  are  the  impediments  to  perceiving  what  is 
necessary? 

What  impediments  might  affect  the  user’s  ability  to  act 
appropriately? 

What  provisions  should  there  be  to  alert  the  operator  when 
something  needs  attention? 
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A  Sample  Use  case 


Source 

Platform 

Available  Data 

ETA 

NIC 

UAV 

High  Resolution  IR 

5  min 

US-12 

Platoon 

Visual 

30  min 

NOR-2 

Satellite 

1  m  Multispectral 

3  hours 

CAN-3 

Special  Forces 

Medical  Condition 

1  hour 

UK-1 

F-16 

Sighting 

-10  min 

FRA-4 
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